On Multi-Layered Connectionist Models: Adding Layers vs. Increasing Width

In this paper, we explore the computational potential and limitations of the multi-layered con-nectionist models [Minsky and Papert, 1968]. We found that the number of layers and the width are two crucial parameters for the multi-layered connectionist models. If each layer has the same size n and we increment the number of layers by 1, then the number of problems solved will increase 0{n 3) times. On the other hand, suppose the number of layers is equal to 2. If we increment the width by 1, then the number of problem solved will increase <D(n n) times, where n is the input size. Hence, we can extend a 2-layered connectionist model by adding layers or increasing width. Our conclusion is that increasing width is better than adding layers. 1 Introduction For studying learning in the multi-layered connectionist models, It is important to understand the computational potential and limitations of the multi-layered connec-tionist models. The single layered connectionist models have limited computational ability. For example, there is no single layered connectionist machine which can compute the 'exclusive or' of two bits. But the 'exclusive or' problem can be solved by a 2-layer connectionist machine. However, the computational ability of 2-layer con-nectionist models depends on the width of the models. We define the width of a connectionist machine to be the maximum number of neurons in one layer of the connec-tionist machine. The number of layers and the width are two crucial parameters for the multi-layered connection-ist models. In this paper, we examine two extreme cases of the multi-layered connectionist models. In the first case, we fix the width of the multi-layered connectionist models and see what happen when the number of layers is increased. In the second case, we fix the number of layers to be two and see what happen when the width is increased. Our result is that in the first case, if each layer has the same size n and we increment the number of layers by 1, then the number of problems solved will increase 0(n n) times; in the second case, when we increment the width by 1, the number of problem solved will increase 0(n n) times, where n is the input size. 2 Layered Connectionist models A layered connectionist machine is a special case of con-nectionist models. It has t layers and one input layer (layer zero). The input (bottom) layer contains n input …